Skip to content

Pruning should affect candidate datasets too - #10877

Merged
jmpesp merged 12 commits into
oxidecomputer:mainfrom
jmpesp:prune_more
Aug 6, 2026
Merged

jmpesp merged 12 commits into
oxidecomputer:mainfrom
jmpesp:prune_more

Conversation

@jmpesp

@jmpesp jmpesp commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

A few weeks ago we were debugging an exhaustive search through all local storage allocations, and determined it was caused by the invalid PgNumeric conversion to ByteCount. The next step was to cause that search to bail out in order to unwind the saga: this was supposed to be achieved by adjusting size_used column of the rendezvous local storage dataset to a number that would not be affected by the invalid conversion, and we did this, but observed that the search continued anyway!

Consider the following incomplete allocation list:

    0: allocation0 -> zpool0
    1: allocation1 -> zpool1
    2: allocation2 -> zpool2
    3: allocation3 -> zpool3
    4: allocation4 -> zpool4
    5: allocation5 -> zpool5
    6: allocation6 -> zpool6
    7: allocation7 -> zpool7

Note the allocations and zpools here are ordered to make the example easier but will actually be random depending on the order of rows returned.

The remaining allocations to perform are 8 and 9, and they have a list of zpools to try to allocate against. The pathologically bad case is if any zpool could be a candidate:

    allocation8: [zpool0, zpool1, ..., zpool6, ..., zpool8, zpool9]
    allocation9: [zpool0, zpool1, ..., zpool6, ..., zpool8, zpool9]

Previous to this commit, the pruning step only affected the list of incomplete allocations. This doesn't cause the search to bail out! If zpool6 runs out of space due to concurrent allocations, the pruning step will not retain incomplete allocations that use zpool6:

    0: allocation0 -> zpool0
    1: allocation1 -> zpool1
    2: allocation2 -> zpool2
    3: allocation3 -> zpool3
    4: allocation4 -> zpool4
    5: allocation5 -> zpool5
    6: allocation6 -> zpool6 <- any search that made it this far
    7: allocation7 -> zpool7    will be pruned

but the iterator will continue to descend down the search space using zpool6 because it's still in the candidate list:

    0: allocation0 -> zpool0
    1: allocation1 -> zpool1
    2: allocation2 -> zpool2
    3: allocation3 -> zpool3
    4: allocation4 -> zpool4
    5: allocation5 -> zpool5

    allocation6: [zpool0, zpool1, ..., zpool6, ..., zpool8, zpool9]
                                         ^- still in the list!

This commit adds a pruning step that removes from an allocation's candidate list based on the updated ZpoolGetForSledReservationResult information.

A few weeks ago we were debugging an exhaustive search through all local
storage allocations, and determined it was caused by the invalid
PgNumeric conversion to ByteCount. The next step was to cause that
search to bail out in order to unwind the saga: this was _supposed_ to
be achieved by adjusting `size_used` column of the rendezvous local
storage dataset to a number that would not be affected by the invalid
conversion, and we did this, but observed that the search continued
anyway!

Consider the following incomplete allocation list:

        0: allocation0 -> zpool0
        1: allocation1 -> zpool1
        2: allocation2 -> zpool2
        3: allocation3 -> zpool3
        4: allocation4 -> zpool4
        5: allocation5 -> zpool5
        6: allocation6 -> zpool6
        7: allocation7 -> zpool7

Note the allocations and zpools here are ordered to make the example
easier but will actually be random depending on the order of rows
returned.

The remaining allocations to perform are 8 and 9, and they have a list
of zpools to try to allocate against. The pathologically bad case is if
any zpool could be a candidate:

        allocation8: [zpool0, zpool1, ..., zpool6, ..., zpool8, zpool9]
        allocation9: [zpool0, zpool1, ..., zpool6, ..., zpool8, zpool9]

Previous to this commit, the pruning step only affected the list of
incomplete allocations. This doesn't cause the search to bail out! If
zpool6 runs out of space due to concurrent allocations, the pruning step
will not retain incomplete allocations that use zpool6:

        0: allocation0 -> zpool0
        1: allocation1 -> zpool1
        2: allocation2 -> zpool2
        3: allocation3 -> zpool3
        4: allocation4 -> zpool4
        5: allocation5 -> zpool5
        6: allocation6 -> zpool6 <- any search that made it this far
        7: allocation7 -> zpool7    will be pruned

but the iterator will continue to descend down the search space using
zpool6 because it's still in the candidate list:

        0: allocation0 -> zpool0
        1: allocation1 -> zpool1
        2: allocation2 -> zpool2
        3: allocation3 -> zpool3
        4: allocation4 -> zpool4
        5: allocation5 -> zpool5

        allocation6: [zpool0, zpool1, ..., zpool6, ..., zpool8, zpool9]
                                             ^- still in the list!

This commit adds a pruning step that removes from an allocation's
candidate list based on the updated ZpoolGetForSledReservationResult
information.
@jmpesp
jmpesp requested review from hawkw and smklein July 21, 2026 15:58
info!(self.log, "removed {removed} from queue");
}

removed = 0;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Can we use a different variable name here? I'd rather these be distinct, rather than trying to rely on "resetting" a single value with two different meanings

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, done in 0dc86bb

});

if removed > 0 {
info!(self.log, "removed {removed} from queue");

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
info!(self.log, "removed {removed} from queue");
info!(self.log, "removed allocation lists from consideration during instance provisioning"; "count" => removed);

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

changed to be less vague in 0dc86bb, but I didn't separate out the count, I'd rather that be part of the message

}

if removed > 0 {
info!(self.log, "removed {removed} from candidate datasets");

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
info!(self.log, "removed {removed} from candidate datasets");
info!(self.log, "removed candidate datasets from consideration during instance allocation"; "count" => removed);

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also 0dc86bb

if removed > 0 {
info!(self.log, "removed {removed} from candidate datasets");
}
}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you think about re-assessing a more holistic check of "can this set fit" here?

There are a couple checks we perform when initially assembling the CompleteLocalStorageAllocationsList:

"Are there enough pools to satisfy this request"
https://github.com/jmpesp/omicron/blob/5886e0028363594e435a8fdaa1fdc01ba787cfd2/nexus/db-queries/src/db/datastore/sled.rs#L465-L477

"Are there any remaining candidate datasets"
https://github.com/jmpesp/omicron/blob/5886e0028363594e435a8fdaa1fdc01ba787cfd2/nexus/db-queries/src/db/datastore/sled.rs#L505-L516

We could presumably re-check these constraints here, with something like:

  let distinct_pools: HashSet<ZpoolUuid> = self
      .allocations_to_perform
      .iter()
      .flat_map(|r| r.candidate_datasets.iter().map(|c| c.pool_id))
      .collect();

  let infeasible = self
      .allocations_to_perform
      .iter()
      .any(|r| r.candidate_datasets.is_empty())
      || distinct_pools.len() < self.allocations_to_perform.len();

  if infeasible {
      self.queue.clear();
  }

Without doing this, I think these cases of "you asked for N disks, but we only have M < N zpools" will be doing a significant amount of work in the invocation of next() within CompleteLocalStorageAllocationLists. Internally, the self.queue.pop() in that next implementation will need to get exhausted before returning, which seems expensive.

For example:

  • Suppose we're in that while let Some(incomplete_allocation_list) = self.queue.pop() loop, and we've made k projected allocations thus far
  • For the next request k, we loop over candidate_datasets, keeping the ones that are still in candidates_left (which, in this case, would be M - k pools). For each of those M-k pools, we push new possible IncompleteAllocationLists.

Because we have a "loop, pushing children (and doing clones) for each candidate_dataset", I think this is a factorial space exploration before we exhaust the set of candidate allocations.

As an aside: We might want to re-evaluate this whole approach here, and consider a more efficient bipartite matching algorithm to avoid having a worst-case-factorial search here in the first place? I think my proposal would help in cases where we have an instance requesting N homogenously-sized disks, but I'm not sure it works if the local storage disks are differently-sized.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you're right that the iterator logic will spin if someone asked for N disks but we only have M < N zpools, but there's already a Not enough zpools to satisfy the number of allocations required check for that in the iterator's new. I added a test that exercises this in 511cd30

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So the check here covers this issue at construction-time, but my concern is about that infeasible state arising mid-search.

The new pruning logic added in this PR can change the candidate lists into a state that our constructor would have rejected, but it doesn't get a chance to.

Specifically:

  1. If candidate_datasets becomes empty (which we check in the constructor here), or
  2. The number of distinct pools across all candidate lists is now smaller than the number of requests (which we normally check in the constructor here)

If either of these happen as an outcome of prune_invalidated_allocation_lists, we never re-check them. I'm mostly worried about wasted work in https://github.com/jmpesp/omicron/blob/367f71137c72d64ab13ad8400a0cbe377f24eb27/nexus/db-queries/src/db/datastore/sled.rs#L775

NORMALLY I wouldn't care much; our checks should identify these cases and eliminate them anyway, but the next function in impl<'a> Iterator for CompleteLocalStorageAllocationLists<'a> is an extremely expensive operation: unless the allocation has already succeeded, it's generating quite a bit of additional queue work.

I made a separate comment below about the ..._pruning_{1,2} tests passing the wrong instance IDs - but I tried to check out this branch, fix that, and then compare the cost of "how expensive it it to call CompleteLocalStorageAllocationLists::next() in the "pruning_2" test with / without the check I suggested?

  • The PR, as-is (with the correct instances patched): next() takes 3.34 seconds of pure CPU time.
  • With the mid-search re-check I suggested: next() takes 979 nanoseconds, because it's not spending time throwing things into / out of the queue.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will also admit: If we think we're going to re-do this whole approach in the near future, we can toss this optimization. We don't need to refine this if we'd prefer to replace the exhaustive search approach instead.

affinity: None,
ncpus: 2,
memory: external::ByteCount::from_gibibytes_u32(16),
disks: (0..10)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the pessimistic case I described above could be triggered by changing this line to:

Suggested change
disks: (0..10)
disks: (0..1)

Basically:

  • Have a request to provision many disks
  • Concurrently, eliminate one disk's viability
  • Observe that walking the remaining search space shouldn't take a terribly long time.

(FWIW, I think this could be a companion test to this existing one, if you want to still test both behaviors: one where the concurrent request consumes all U.2s, and one where it only consumes one of them)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we could probably even be proptesting across all possible local disk counts, although i don't know if that would really get us anything more useful than just having 0..1 and 0..10 test cases

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah I think the trickiness here is that we actually do handle both cases, but I think the performance profile is the thing that varies significantly between them

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added this test in 4b2c631, it finished in about 10 seconds on my workstation

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hm, i wonder if we ought to be running these as benchmarks so that we can spot regressions in performance. not a blocker for this PR, since I think that may require some unrelated infrastructural work.

Comment on lines +613 to +632
match zpools_for_sled.get(&candidate_dataset.pool_id) {
Some(zpool_for_sled) => {
// Zpool still exists, does it still have room?

if !zpool_for_sled.has_room_for_allocation(
allocation.request.required_dataset_size(),
) {
// Do not retain this as a candidate dataset, it no
// longer has room.
removed += 1;
return false;
}
}

None => {
// Zpool doesn't exist anymore!
removed += 1;
return false;
}
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

style nit, take it or leave it: we could maybe reduce "rightward drift" by one level of indentation here by writing:

Suggested change
match zpools_for_sled.get(&candidate_dataset.pool_id) {
Some(zpool_for_sled) => {
// Zpool still exists, does it still have room?
if !zpool_for_sled.has_room_for_allocation(
allocation.request.required_dataset_size(),
) {
// Do not retain this as a candidate dataset, it no
// longer has room.
removed += 1;
return false;
}
}
None => {
// Zpool doesn't exist anymore!
removed += 1;
return false;
}
}
let Some(zpool_for_sled) = zpools_for_sled.get(&candidate_dataset.pool_id)
else {
// Zpool doesn't exist anymore!
removed += 1;
return false;
};
// Zpool still exists, does it still have room?
if !zpool_for_sled.has_room_for_allocation(
allocation.request.required_dataset_size(),
) {
// Do not retain this as a candidate dataset, it no
// longer has room.
removed += 1;
return false;
}

IMO that reads a little nicer because the logic flows top-to-bottom instead of kind of middle-out?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

personally I like the way it reads with the match :)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

well, you're wrong, but okay! :)

affinity: None,
ncpus: 2,
memory: external::ByteCount::from_gibibytes_u32(16),
disks: (0..10)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we could probably even be proptesting across all possible local disk counts, although i don't know if that would really get us anything more useful than just having 0..1 and 0..10 test cases

@hawkw hawkw left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this looks pretty good! I'd want a +1 from Sean as well, though.

.internal_context("selecting multiple disks failed")
})?
.into_iter()
.map(|disk| (disk.id(), disk))

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this makes me feel like the HashMap ought to be an iddqd::IdHashMap (or IdOrdMap), with an IdHashItem implementation for db::model::Disk, so that the invariant that the key is always the disk ID can be inherent in the data structure rather than implicit. not a huge deal since this hash map is relatively short lived and isn't mutable once it's collected, but it still feels a bit tidier.

Comment on lines +607 to +612
// Is it possible the disk has been hard-deleted somehow?
// Otherwise how would we land here, given that we just created
// the map!
return Err(Error::internal_error(&format!(
"disk id {disk_id} not found in map"
)));

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hm, i wonder whether we ought to be handling this by logging a warning and treating it as equivalent to the disk being deleted?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

right now, disks shouldn't ever be hard-deleted, so I think this should be a 500 level error.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that's fair, i was just thinking about defensive programming in case someone or something is messing with the database manually.

Comment on lines +613 to +632
match zpools_for_sled.get(&candidate_dataset.pool_id) {
Some(zpool_for_sled) => {
// Zpool still exists, does it still have room?

if !zpool_for_sled.has_room_for_allocation(
allocation.request.required_dataset_size(),
) {
// Do not retain this as a candidate dataset, it no
// longer has room.
removed += 1;
return false;
}
}

None => {
// Zpool doesn't exist anymore!
removed += 1;
return false;
}
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

well, you're wrong, but okay! :)

@morlandi7 morlandi7 added this to the 22 milestone Aug 5, 2026

impl ByteCount {
// None of these three constructors can create a value larger than
// `i64::MAX`. (Note that a `from_tebibytes_u32` could overflow u64.)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Your additions to this file inviolate this comment, and allow overflow?

Do we truly need this change? It seems unsafe

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oof, nice catch. Removed in f595927

CompleteLocalStorageAllocationLists::new(
&logctx.log,
config.sleds[0].sled_id,
config.instances[0].id,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure I understand what's happening in this test.

  • instance_2_local_storage_disks lists disks from config.instances[1].id
  • ... but this constructor uses config.instances[0].id ?

I think this means that the disk attachment check (our in-prod code) at the top of prune_invalidated_allocation_lists calls self.queue.clear() and doesn't actually check what this PR is trying to check.

I think local_storage_allocation_iterator_pruning_2 is doing the same thing below (gathering disks from instance 1, then creating a CompleteLocalStorageAllocationLists for instance 0).

Should these instances match?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes! this is an awful typo, fixed in 78381fd

if removed > 0 {
info!(self.log, "removed {removed} from candidate datasets");
}
}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So the check here covers this issue at construction-time, but my concern is about that infeasible state arising mid-search.

The new pruning logic added in this PR can change the candidate lists into a state that our constructor would have rejected, but it doesn't get a chance to.

Specifically:

  1. If candidate_datasets becomes empty (which we check in the constructor here), or
  2. The number of distinct pools across all candidate lists is now smaller than the number of requests (which we normally check in the constructor here)

If either of these happen as an outcome of prune_invalidated_allocation_lists, we never re-check them. I'm mostly worried about wasted work in https://github.com/jmpesp/omicron/blob/367f71137c72d64ab13ad8400a0cbe377f24eb27/nexus/db-queries/src/db/datastore/sled.rs#L775

NORMALLY I wouldn't care much; our checks should identify these cases and eliminate them anyway, but the next function in impl<'a> Iterator for CompleteLocalStorageAllocationLists<'a> is an extremely expensive operation: unless the allocation has already succeeded, it's generating quite a bit of additional queue work.

I made a separate comment below about the ..._pruning_{1,2} tests passing the wrong instance IDs - but I tried to check out this branch, fix that, and then compare the cost of "how expensive it it to call CompleteLocalStorageAllocationLists::next() in the "pruning_2" test with / without the check I suggested?

  • The PR, as-is (with the correct instances patched): next() takes 3.34 seconds of pure CPU time.
  • With the mid-search re-check I suggested: next() takes 979 nanoseconds, because it's not spending time throwing things into / out of the queue.

@jmpesp
jmpesp enabled auto-merge (squash) August 6, 2026 20:31
@jmpesp
jmpesp merged commit 066db5c into oxidecomputer:main Aug 6, 2026
19 checks passed
@jmpesp
jmpesp deleted the prune_more branch August 6, 2026 21:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants